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BEVIEW OF CBP INTERVIEW PROCEDURE AND DATA 
Penelope Wood 

ABSTRACT 

The interview procedure used with educable mentally 
retarded children (EMR) in the evaluation of media at the 
Computer Based Project is described> and the findings of 
an examination of the interview data collected are reported. 
The interview procedure utilizes questions designed to 
indicate whether the child had visual and auditory exper- 
iences while exposed to a film, and the relative strength 
of these experiences. The interviewer also makes a quanti- 
tative assessment of the child's verbal capahilitles . An 
ex post facto analysis of the data for three films revealed 
a modei^ately high correlation between verbalization ratings 
and sum word-counts for the interview questions* Overall, 
the data supported findings which indicate that the verbal 
performance of El^lR children is limited^ The implications 
for learner-based evaluation systems are discussed. 
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SPECIAL REPORT No. 7321 
COMPUTER-BASED PROJECT for the EVALUATION 
of MEDIA for the HANDICAPPED 

T i t I 8 i REVIEW OF CBP INTERVIEW PROCEDURE AND DATA 

BY: Penelope Wood 

The Computer Based Project for the Evaluation of Media for the Handicapped, 
based on contract SOEC-9-'423C17-4357 (616) between the Syracuse (N-Y.) City School 
District and the Media Services and Captioned Films Branch, Bureau of Education 
for the Handicapped {United states Office of Education) for the five year period 
July 1, 1969 through June 30, 1974» The major goal is to improve the instruction 
of handicapped children through the development and use of an evaluation system 
to measure the instructional effectiveness of films and other material?=3 with 
educable mentally handicapped (EMH) children, in-service training and media support 
for special teachers, and studies related to the evaluation process and the 
populations used. 

The Project has concentrated on the 600 films and POO filmstrips from the 
Media Services and Captioned i^i 1ms (BEH - USOE) depository; however, specific 
packages from Project LIFE, various elementary math curricula, and selected 
programs from Children's TV Workshop have also been evaluated. The evaluation 
model used requires that! 1) objectives of materials be specified and written; 

2) instruments be constructed to test and measure effectiveness r and i 3) children 
bo the major sources of evaluation information. A number of instruments and 
methodologies are employed in the gathering of cognitive and affective data frcm 
900 EMH children and 80 special teachers to make the effectiveness decisions . 
Over half of the EMH population can neither read or x>;rite; therefore, a unique 
Student Response System (SRS) is employed, consisting of a twenty station G*E.- 
1000 SRS which can be operated in a group or individual recording mode and is 
connected to a remote computer system. The computer capabilities consist of 
remote telephone connections to the Rome (£^.Y.) Air Development Command, the 
lloneyvell time-shared network, and the Schenectady (N.Y.) G E Research and 
Development Center? and batch mode capabilities of the Syracuse City Schools, 
Syracui>c University, and various commercial sources. 

In-servico and media support activities provide on-the-job training for 
teachers, teacher aides, equipment^ and materials to the special teachers in 
the city schools. The research activities havo centered around investigations 
and special problems related to the development of the evaluation model. The 
four major areas considered aret 1) testing effects, 2) captioning effects, 

3) special student characteristics; and, 4) evaluation procedures validation. 

Documentation of the major activities appear in the five annual reports 
and the 600 evaluations prepared on materials used. Staff members v/ere oncouragod 
to prepare special reports and the attached paper is one of these. The opinions 
expressed in this publication do not necessarily reflect the position or policy 
of the Computer Based Project, the United States Office of Education, or the 
Syracuse City School District, and no official endorsement by any of the agencies 
should be inferred. 
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Review of CBP Interview Procedure and Data 

The Computer Based Projectfor the Evaluation of Media for the 
Handicapped, funded by the United States Office of Education, Bureau 
of Education for the Handicapped, and under contract \/lth the Syracuse 
City School District, Department of Special Education, has developed a 
learner-based model for the evnl uat Ion of media. The model ts a multi- 
step process involving: (1) ascertainment and assignment of Instructional 
objectives for a given material ; (2) establ ishment of levels of performance 
for various student populations and data col loct Ion , us I ng student populations, 
on cognitive, attent lonal , and attltudlnal responses resulting from media 
exposure; (3) field trial of the materirsl by a classroom teacher in a 
classroom setting. Much act i vl ty of the Project Is concerned wl th use of 
this model to evaluate the instructional effectiveness of materials 
produced for deaf chl 1 dren , espec ially capt ioned fi 1ms ,wi th educable 
mentally retarded (EMR) and emot lonal ly disturbed (SSR) chl Idren. 

The Project Is now completing its fourth year of operation. In 
accordance wl th contract obi igat Ions , there has been cont Inuous data 
col lect Ion and assessment of media both pr lor to - and simultaneously v/l th ^ 
cont I nu I ng development > evaluat ion > and ref Inement of model procedures . In 
theory, a learner-based model for evaluation of i nstruct lonal materials ^ ^ ; 
may draw considerable merit from the ass'jmptlons of learner-centered education* 
In practice, th^ collection c^nd analy:?:ln^ of lorrner-response d^ta moy be 



fraught v/lth questions of validity and reliability, and these must be 
continually examined. The model comprises several subsystems for data 
collection; It Is the student Interview procedure and data which are the 
concerns of this report • 
Interview Proc e dure 

The Interview procedure was designed '^to acccmnx5date both the 
desire to give Che child an opportunity to give a free response and the 
need to standardize the Items for analysis purposes'^ (Bond, p.8i). Immediately 
following a film presentation, a student Is selected from the group by seat 
number and taken by the Interviewer to a quiet place. The Items of the 
Interview are then asked by a staff member and answers recorded verbatim 
onto the intervlev/ form. Interviewers are instructed ''to create an open 
dialogue with the child and not to pressure a response" (Bond, p.Bk). Specific 
questions are asked in attempt to guide the chl Id In te! 1 1 ng the Interviewer 
what he experienced dur Ing the viewing. Items are designed to Indicate 
whether the chl Id had visual and auditory experlcncesvvhl 1 o exposed to the 
film and the relative strength of these experiences. Followl ng the Interview 
the interviewer makes a quant I tat I ve asfiessment of the chtld^s verbal capabilities 
on a graphic scale of 1 to 6. The graphic rating scales give an Interpretative 
rat lo of three performance characterlst Ics of the chl 1 d : verbal I z-^t Ion, 
verbalization of visual experiences; and verbalization of audi tory experiences . 
Such assessments are made in attempt to obtain a measure for comparing a 
child's resportse against his capability for making a response. Subsequently , 
interview reponses are encoded quantitatively for computer processing. Answers 
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to opef^-ended questions are recorded as the number of words In the response, 
coded 1 through 9 with a^, and the omi tted : a child's response , '•making 
doughnuts," to the question, ^Vhat do you think the film was about?" Is 
encoded as 2* The several closed quest Ions call Ing for yes or no^ answers 
are encoded as 1 and 2, respectively. 

The Interview procedure as used by Computer Based Project has yet to 
be validated and assessed for rel tab! II ty. Given the comnunicat Ive problems 
of mentally retarded and emotionally disturbed children, there Is some 
question as to v;hether or not a verbal Interview measures effectively and 
consistently the perceptual -cogn I t Ive responses of a child to a film, 
Schlanger (l953)j measuring verbal productions of primary EHR children, 
reported evidence attest /ng their being verbal ly handicapped. Such 
children demonstrated a shorter r 'ean length of sentence than normal children, 
twenty-one percent of the sentences one vA:)rd In length* These Sentences 
general ly i nvol ved the nam 1 ng of objects and people, and Schlanger has ct ted 
the use of such concrete-noun sentences as further evidence of 1 Ingulstlc . 
Immaturl ty* Wood (I960) has described communication proMerns of EMR <fhl Idren 
fn her studies of learning potent lal . Evans (1965) reported erratic verbal 
behavior among mentally retarded children who, given an opportunf ty to 
communicate wi th the Investigator, demonstrated low to mldllng probability 
of cooperat Ion. Anas t as lo (1972) , working wl th the Computer Based Project , 
observed the lack of verbal expression In E^^f^ and SSR children. He noted 
discrepancy between the observed behavioral responses of children while 
watching a film and their Interview-reported responses. 



In addition to the basic question of whether or not a verbal procedure 
can be Judged valid and reliable to measure a verbally handicapped child's 
response to iDedia, there are questions of validity to be axamlned In the 
encoding of student responses for data analysis. Assuming the open-ended 
questions are asked for the purpose of el letting content from a student i 
there Is considerable debate as to whether a quanti tatlve measure of the 
verbal response has any validity as a measure of content; whether the 
number of words used to answer a question has any significance for what 
has been said In an Interview, Little research was found deal Ing \/l th the 
quantitative encoding of verbal responses as used by the Computer Based 
Project. Schlanger (1953) measured length of verbal product Ions of mentally 
retarded children and considered such measures useful a a test of language 
development, but made no attempt to equate sentence- length with content. 
Jordan, reviewing the 1 1 terature of language and mental retardatlonj found 
word-count a ^-relat I vely unused measure whose correlates do not seem to 
have been fully explored** (1967) • Sause and Crowley (1970) describe a 
computer technique for analyz I ng I ntervlews wl th children; however, their 
approach Involves a content analysis through Identification of key words and 
I s not a word-count . 
Data Examination 

To begin analys Is of validl ty and rel labll I ty questions, the Investlgato 

examined the Interview data held by the Project. Two hypotheses were 

formula tody comparing verbal per fo rmance against es t f ma ted verbal capabi I f ty : 

(1) An £MR chlld^s verbal behavior within the CBP fntervfew 
situation, as measured by the verbal I zat Ion score assigned 
by the interviewer > remains relat ively cons tant from one 
f i Im/lntorvievr to another. 



(2) An EMR child's verbal behavior wl thin the CBP Interview 
situation, as measured by the sum word-count of questions 
I >3>5i7>8i I'+i varies from one f I Im/lntervlew to another. 

The Investigator reasoned that If the verbalization score assigned by the 

interviewer Is a val Id and reliable measure of a child's capability for 

making verbal responses, that score should remain relatively constant/ If on 

the other hand, quantltltlve measures of responses to open'-ended questions 

have any relationship to a child's experience with one film differs from his 

experience v/lth another, then the sum word-count wl 1 jshow variance from one 

film Interview to another. 

To test the above hypotheses, an ex post facto Inves 1 1 gat Ion of data 

was planned using three different films as the Independent variable and 

analyzing variance in verbalization scores and word-counts. Interview data 

for spring 1972 were pulled from the data banks. Several different interview 

forms had been used; Form 8, the latest adopted and most standardized, was 

selected for analysis, and cases were found listed. At this point , 

however, the planned Investigation appeared questionable, s Ince Interviewers 

who had collected the data were not 1 I sted In the data< Considering 

Interviewer a critical variable for both the stated hypotheses, the investigator 

returned to the media files, drew names of Interviewers from all available 

original Interv lew forms , and coded these Into the data. Cases were then 

reviewed. To test hypotheses (1) and (2) , cases in which one chl Id was Interviewed 

successively by the same interviewer for different films were sought, but of 

the total , only V such cases v/ere founds Cases In which randomly selected 

students of one level v/ero i nterv lewed by one or two Interviewers for 

successive films were toe few for meaningful analysis, Hypotheses (0 and 
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(2) could not be tested with the available data* 

Data was then examined for any possible usefulness. A third hypothesis 

having some bearing on questions of validity and reliability was formulated: 

The sum v/ord-'count from questions 1 ,3i5»7#9il'' of 
the CBP Interview form, as completed by EMR students, 
correlates posi tivoly with the verbal Izatlon score assigned 
by the interv iewer . 
The Investigator reasoned that there would be a high positive correlation 

between verbal ! zat Ion scores and sum word-counts . An extremely high 

correlation, however, would suggest too high agreement In scores which 

purportedly measure different things: general capability for verbal 

expression and verbal performance In a specific situation. Of nine 

Interviewers three had conducted roughly half of the I ntervlews , and these 

wore selected for analysis. Data v;erc transferred to format suitable for 

SPSS processing and programs run for correlation and cross-tabluat Ion. 

As predicted, a high correlation was found between the verbal Izatlon 
rated assigned by the interviewer and the sum wo rd*-count for open-ended 
questions of the Interview. The Pearson coeff Ic lent for the two variables 
on 181 Interview cases conducted by Interv lev/e'rs 3 ,5iand 7 Is +,65* 

Cross-tabulations were run as follows; intervlewer^by sum word-county 
student level (primary, iMtermedi ate, secondary) by sum word-count , f Mm by 
sum word'-count , verbal izat Ion rating by sum v:ord-count; film by sum word- 
count con troll I ng for level, f I Im by sum word^count control 1 1 ng for 
Interviewer , and student level by sum word-count control 1 Ing for Interviewer. 
Sum word-counts v/erc grouped 0 to 5» (> to 10.... 21 to 30 and recoded. Since 
27 films and 116 different students were involved, however, the data have 
little value for suggesting either differences among Interviewers or 
differing effects of film as measured by Interview, in the major! ty 
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of cases, sum word-counts of Interviews r^inged from 6 to 15 words: for 
Interviewer 3, 72^; Interviewer 5i 56%; Interviewer 7» 55%. Differences In 
cross-tabulation of sum word-count by film, controlling for student levels 
are too scattered to be meaningful, even assuming Interviewer rel lability. 
Cone 1 us Ions 

The moderately high posi tf ve correlat Ion found between verbalization 
rating and sum word-count suggests some Implications concerning the 
procedure. Verbalization scores are assigned by the lntervlev;ef fol lowing 
the Interview and are Intended to represent a sti'dent's capability for 
verbal response as opposed to actual performance. There Is danger In such 
a procedure that the estimated capabl 1 1 ty be simply a reflection of a 
specific performance; that the verbal Izatlon score assigned by simply a 
scalar measure of how nany words a student has spok^^n In a given Interview. 
An extremely high correlation between the verbalization rating and the 
sum word-^count would suggest 1 1 ttle difference between the two scores and 
the verbal Izatlon c-^pabM ity rating meaningless. The moderately high 
correlation Is a pred 1 ctable est Imate for scores which are related b^Jt 
(by stated assumption) not equivalent. Cross-tabulation data show low 
verbalization scores of 2 and 3 assigned to Interviewees performing wl th 
sum word-counts which vary from lowest (O to 5 words) to highest (21 to 30) 
range* Thus t t may be stated that the two scores are not redundant. 
Whether the variance between verbal capabil I ty and performance Is related 
to film, however, cannot be determined from the data. 

m-::-:M' One conclusion drown from the data may be significant for the worf^of 
the Project! for Interviews an^ilyzed represent Ing a) 1 three student levels, 



the sum word-count for 6 open-ended questions totals 10 words or less In 
'i^t of the cases, 15 words or less In 73?; of the cases. The I Iml ted verbal 
performance Is In keeping with the findings of Schlanger, Evans , and 
Anastaslo, attesting the verbal limitations of EMR children. When the 
response-mode Itself Is so limited, It Is quest lonable whether that mode 
can be used effectively to register differing experiences with film; 
and the questions raised make al I the more critical the need to val Idate the 
procedure ♦ 

The stated purpose of the interview procedure Is Important for a 
learner-based evaluation system and connblnes two critical elements: the 
opportunity for the child to give a free response about his experience wl th 
film or other InstructlonaT material ; and the standardization of 1 tems f c r 
analysis purposes. It Is posslb)<^ that the procedure couM be refined and 
that a word-count measur^^ to open-ended questions could be found to have 
!?ome relationship to a chMd^s differing experiences with one film or 
another. Furthermore, such a finding would have value for learner-based 
evaluation models beyond the work of Computer Based Project* However, the 
data examined have ' I ttle value for determining a g Iven student 's response 
to film, the film^s effect iveness wl th a group of s tudents , or even the 
validity of the procedure. It Is suggested that the Project e 1 ther suspend 
Interview data col lect ion or revise the procedures, control I i ng for Inter- 
viewer. If the interview procedure is continued, It is sugcjested that a 
research study comparing student responses In varying question modes - open- 
ended quest ions, closed questions , pictorial ly presented quest Ions - be 
conducted to assess the relationship of word-count during an interview with 
Information about a child's interaction with film obtained through mro MM:- 
concrete procedures. 
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